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ABSTRACT 

A report on implementation of the Multicultural 
Reading and Thinking (McRAT) Program in Arkansas from 1989-92 
chronicles the program's development, operation, and results. McRAT 
is a staff development program designed to help teachers infuse 
higher order thinking processes and multicultural concepts into 
regular classroom curricula (grades 3 to 8) . The program provides 
teachers with explicit strategies for teaching reasoning and problem 
solving throughout the curriculum; it trains teachers in state~of _ the 
art alternative assessment approaches to evaluating student progress, 
emphasizes intercultural concepts as a meaningful context for 
application of thinking and problem- solving strategies, and includes 
methods for integrating direct instruction with other effective 
strategies such as cooperative learning, process writing, and 
facilitated group discussion. Data on the program were from 
evaluation of student essays using scoring consistent with those of 
other large-scale writing assessments. Findings included the 
following: (1) McRAT students demonstrated evidence of higher order 
thinking through significantly higher gains on analytically scored 
essays than non-McRAT students; (2) participants demonstrated 
retention of learning over time; (3) McRAT students outperformed 
control students regardless of classification; (A) no statistically 
significant difference between performance of minority versus 
non-minority students; and (5) assessment procedures showed a high 
degree of inter-rater reliability. Ten tables provide detailed 
program evaluation results. (Contains 11 references.) (JB) 
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Introduction 

This report provides information about 
the development, operation, and results of 
the Multicultural Reading and Thinking 
(McRAT) Program in Arkansas for the pe- 
riod 1989-92. Part I gives explanatory and 
background information on the McRAT 
Program. Part II reports research results on , 
program effectiveness and continuity over 
a three-year period. 

Parti — 

The McRAT Program 

Multicultural Reading and Thinking 
(McRAT) is a staff development program 
designed to help teachers infuse higher- 
order thinking processes and multicultural 
concepts into regular classroom curricula. 
It has been developed, refined, and rigor- 
ously evaluated since its beginning in 1986 
as a collaborative effort of Arkansas De- 
partment of Education reading specialists 
and classroom teachers. 

It has been validated, grades 3-8, by the 
National Diffusion Network (NDN), U.S. 
Department of Education, and was named 
one of the most effective educational in- 
terventions in Arkansas by the 1992 Edu- 
cational Policy Study by the Winthrop 

i£ Rockefeller Foundation. 

McRAT integrates thinking and prob- 

© lem-solving processes into the core of edu- 

^ cationa' practice, namely the interactions 
betw e n teacher and student in regular 

S clas ,-ooms. First, the program provides 
tear no s with explicit strategies for teach- 
ing reasoning and problem solving 
throughout the curriculum. 

Second, it trains teachers in a state-of- 
the-art, alternative assessment approach to 
evaluating students' progress. Third, while 
the program emphasizes the teaching of 
intercultural concepts as a meaningful con- 
text for the application of thinking and 
problem-solving strategies, teachers are 
taught how to use the framework of 
higher-order thinking throughout the cur- 
riculum. 

Finally, McRAT includes methods for 
v integrating direct instruction with other 
^V" effective strategies such as cooperative 
JA» learning, process writing, and facilitated 
^ group discussion. 

Consistent with provisions of Act 236, 
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the program embodies the movement to- 
ward linkage between instructional pro- 
grams and "alternative assessment." 

Further, it empowers teachers to deliver 
effective instruction to the wide range of 
abilities found in regular classrooms — in- 
cluding special education students, Chap- 
ter 1 and gifted students, as well as the 
"average" students. 

Rationale: 

McRAT embodies much of what has 
been learned from contemporary research 
about effective contexts for learning, staff 
development, literacy, and assessment. 
Wells (1990), Graves (19b3), and Anderson 
(1983), among others, have emphasized 
that reading comprehension is enhanced 
by reading a wide range of content, by 
reading strategy instruction, and by the 
explicit linkage of reading and writing. 
National reports on student achievement 
in reading and writing have indicated the 
need for instruction that teaches students 
to reason more effectively about what they 
have read, to defend or elaborate on their 
ideas, and to communicate them in writ- 
ing (Applebee, 1988; Langer, 1990). 

McRAT utilizes the framework of 
thinking skills outlined by Quellmalz 
(1985). Four broad categories of thinking 
skills are included: analysis, comparison., 
inference/interpretation, and evaluation. 
This particular framework was selected 
because it addresses a common body of 
knowledge in philosophy and psychology 
(e.g., Bloom, 1971); it is conceptually clear; 
and it easily generates classroom assess- 
ment data (Stiggins, 1989). 

McRAT incorporates findings from re- 
search on effective staff development (e.g., 
Guskey, 1986). This body of work shows 
that, to change classroom practices, teach- 
ers need extensive training that includes 
modeling of desired practices. Moreover, 
teachers need continuous support from ad- 
ministration, competent and easily ac- 
cessed technical assistance, and clear evi- 
dence that their changes are benefiting stu- 
dents. 

Teacher Inservice Program: 

McRAT's training of teachers in Arkan- 
sas extends over two years. As teachers 
receive training, they simultaneously 
implement the program with students. 



Year 1 includes a total of nine days of train- 
ing beginning with an initial three-day ses- 
sion in August, followed by three addi- 
tional days and a series of half-day sessions 
scheduled throughout the year. Adminis- 
trators are strongly encouraged to partici- 
pate in the initial 3-5 days of training. The 
focus of the training is on explicit teaching 
and infusion of strategies for each of the 
four thinking processes; lesson design and 
construction; and procedures for evalua- 
tion of student writing. Half-day sessions 
provide time for collaboration, curriculum 
evaluation, and lesson development. 

Year II training focuses on more ad- 
vanced techniques for curriculum devel- 
opment, classroom assessment and col- 
laboration. It includes peer coaching, 
where teachers observe each other teach- 
ing McRAT strategies and provide appro- 
priate feedback. 

McRAT Leader Program: 

A six-day leadership training program 
has been implemented to assist school dis- 
tricts with dissemination of the program 
within their districts. Applicants must have 
successful McRAT teaching experience, be 
recommended by the school principal and 
regional McRAT trainer, and demonstrate 
leadership qualities. A total of seven have 
been trained and are providing leadership 
in their respective districts. 

School Participation: 

The program is provided to schools 
upon written request to the Department of 
Education reading office. Participation is 
limited to approximately 150 new teach- 
ers each year due to funding and the num- 
ber of trainers available, Training is pro- 
vided by reading specialists at each of the 
fifteen educational cooperatives. Schools 
provide all instructional materials and 
equipment but receive grants to assist with 
the cost of teacher-release time. 

McRAT Instruction: 

Instruction focuses on four fundamen- 
tal kinds of reasoning — analysis, compari- 
son, inference/ interpretation, and evalua- 
tion — that students can use in all aca- 
demic subjects and that also transfer to 
practical situations. Students learn explicit 
strategies for each kind of reasoning and 
an awareness of how the strategies can 
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transfer across problems. For example, 
they might analyze data in math and sci- 
ence, the social institutions of a culture in 
social studies, the different kinds of litera- 
ture in English or reading, or the cultural 
diversity of the neighborhood where they 
live. McRAT lessons directly address stu- 
dent learning expectations from the state 
curriculum frameworks. 

McRAT uses literature commonly avail- 
able in school libraries and classrooms. The 
program recommends fiction and nonfic- 
ticn that will expose students to other his- 
torical periods and perspectives as well as 
different parts of the world. In the increas- 
ing diversity of our nation, as well as the 
emergence of the "global village," the pro- 
gram helps students learn about others 
through their own eyes. For example, af- 
ter reading several selections about 
homelessness, they may write an essay "in- 
ferring what life would be like if you were 
homeless." They may analyze ways in 
which the actions of a story character re- 
flect the customs, values, and beliefs of the 
culture. They may compare the art, music, 
and literature of various cultures or the 
contributions of historical figures. Lessons 
or units of study often involve parents and 
community. 

Part II — 

Research Report 1989-92 

The following is a description of data 
on the McRAT Program for the three-year 
period between 1989 and 1992. Table 1 
shows growth of the program during this 
period. 

Major research findings are described 
with supporting data on program effective- 
ness and continuity during this period. 
Table 2 shows demographics of students 
in evaluation samples. Data were derived 
through evaluation of student essays that 
employ the four types of reasoning that are 
the focus of the staff development process. 
Analyses of students' essays show in- 
creased ability to reason and to communi- 
cate ideas through writing. Essays were 
scored using procedures consistent with 
those of other large-scale writing assess- 
ments. 

TABLE 1 Teachers in McRAT 
Training 1989-92 



TABLE 2 Demographics of Students 
in Evaluation Samples 



Year 


No. 

Teachers 


No. Students 
Involved 


1989-90 


90 


2250 


1990-91 


126 


3150 


1991-92 


125 


3125 


Total 

mtmm 


341 


8525 



Students 


Toul 
No 


Chap*. 
1 


Cifttd 
/Talented 


Minority 


19W-90 
McRAT 


234 


25% 


15% 


32% 


Compmruon 


101 


10% 


15% 


.10% 


1990-91 
McRAT 


10* 


»% 


17% 


22% 


19*1-92 
McRAT 


346 


12% 


20% 


21% 



Finding 1 

McRAT Students demonstrated evidence 
of higher-order thinking as shown by 
significantly higher gains on analyti- 
cally scored essays titan non-McRAT 

students. Results are consistent 
regardless of students' gender, race, or 
prior achievement level. 



1989-90 (Year 1): 

A study was conducted involving 342 
randomly selected fourth, fifth-, and sixth- 
grade students in six schools participating 
in the McRAT Program. The schools se- 
lected were representative of the state 
population in different regions of the state, 
and each was able to provide classrooms 
for both McRAT and control groups within 
the school. 

Table 3 displays the means for McRAT 
and comparison groups. As can be seen 
from these statistics, the McRAT group ini- 
tially scored lower than the comparison 
group on all four essays. 

The post-test means demonstrate a 
clear difference between McRAT and com- 
parison students. McRAT classes signifi- 
cantly exceed comparison classes on all 
four thinking skills. Perhaps the most dra- 
matic is Evaluation, where initially com- 



parison students exceeded McRAT stu- 
dents overall by almost ten points. On the 
post-test, comparison students had re- 
mained virtually the same, while McRAT 
students had gained over 20 points on 
thethinking skill that is taught last in the 
McRAT sequence because it encompasses 
elements of the three others. 

A considerable proportion (about 30%) 
of students' pre-test essays were "off-task" 
and were given a total score of 0 (zero). 
"Off-task" essays included responses that 
were listings of words or sentences, writ- 
ten about topics unrelated to the task, or 
otherwise unconnected discourse. These 
essays were considered unscoreable. 

Table 4 shows that many mor 2 McRAT 
students were included in this group. 
However on the post-test essay, far more 
comparison students were "off-task" than 
McRAT students. Further examination of 
the "off-task" group revealed that students 
with one or more "off-task" essays were 
more likely to be Chapter 1 students. 

The clear implication of this analysis is 
that educationally disadvantaged students 
are less likely to write connected text with- 
out explicit instruction such as that pro- 
vided in McRAT classrooms. Trur«» pat- 
terns are consistent throughout the ihree 
years of this study. 

Table 4 also displays the post-test 
means of McRAT and comparison students 
who were off-task on the pre-test. As can 
be seen by comparing these means to those 
i n Table 3, these students were comparable 
to their peers after McRAT instruction. The 
increased number of off-task responses 
observed in the fourth-grade control group 
post-assessments occurred in one class- 
room, possibly indicating an environmen- 
tal factor. 



Thinking Skill 


McRAT 


Companion 


N Pre Post Gain 


N Pre Post Gain 


Analysis 


157 .35.7 59.3 23.6 


99 44.9 50.8 +5.9 


Comparison 


119 24 4 51.5 26.9 


83 268 35 5 +8 7 


Inference 


76 24.6 45.5 20.4 


65 31.9 32.1 +0.2 


Evaluation 


114 35.8 55.3 19.5 


99 45.5 44.0 1 5 



Note Thew are meant for 
students with scoreable wmw 
for both pre- and post* 
administration* Remilu for 
student! who were ofMaak on 
the pnr-te*t 

(0 total •core) were computed 
Bepjnteiv Thevanationt in 
nu mber of student! 
(N ) reflect this separation 



Thinking 
Skill 


McRAT 


Comparison 


No Off Task 


Post-Test Mean 


No. Off-Task 


Post-Test Mean 


Pre Post 




Pre Post 




Analysis 


73 1 


544 


6 27 


39.0 


Comparison 


52 1 


43 2 


13 19 


26.0 


Inference 


68 10 


39 7 


.37 37 


32 5 


I valuation 


49 r 


49 6 


5 10 


201) 



Finding 2 

McRAT students demonstrated 
retention of learning. 



This study included a longitudinal fol- 
low-up of students from the 1989-90 study 
and a random sample of students who en- 
tered the program during the next two 
years. The purpose was to ascertain 
whether the effects of McRAT were stable 
over time and replicable. 

1990- 91 (Year 2): 

Table 5 shows a comparison of the 
mean scores of students with and without 
previous McRAT instruction. Scores reveal 
that although the students not having pre- 
vious McRAT instruction showed more 
improvement, the overall post-test scores 
for the two groups were very similar. The 
generally higher pre-test scores for the stu- 
dents with previous McRAT instruction 
indicate that much of the previous year's 
learning carried over into the second year 
for these students. 

1991- 92 (Year 3): 

The results were very similar. Overall, 
the students showed average gain scores 
between 9.4 and 29.3. In general, the post- 
test scores for the students with one year 
of previous McRAT instruction were 



higher than those for students without pre- 
vious McRAT instruction. Also, the post- 
test scores for students with two years of 
previous instruction were generally higher 
than for students with one year of previ- 
ous instruction. 

Together, these two patterns seem to in- 
dicate that students with more McRAT in- 
struction tend to perform better and they 
tend to retain more of what they have 
learned. 



Finding 3 

McRAT students, regardless of 
classification, outperformed control 
students. Substantial gains were 

exhibited by McRAT students 
regardless of the number of years of 
previous instruction. 



of mean scores for Chapter I, Regular, 
Gifted, and Special Education students 
with 0, 1, or 2 years of previous McRAT 
instruction. The significant increase in pre- 
test scores each year for students in each 
group indicates that students, regardless 
of achievement level, are learning and re- 
taining their learning from year to year. 

The McRAT curriculum seems to be ef- 
fective for all students, as substantial gains 
were exhibited by all four classifications, 
regardless of the number of years of pre- 
vious instruction they may have had. 



Finding 4 

There were no statistically signifi- 
cant differences between perfor- 
mance of minority vs. non-minority 
students in the McRAT Program. 



Tables 7 and 8 present comparisons of 
mean scores of students who were classi- 
fied as Chapter 1, Gifted, Special Ed, or 
Regular. As expected, Chapter 1 and Spe- 
cial Ed students generally had mean scores 
which were slightly lower than the regu- 
lar students, whose scores were in turn 
slightly lower than those students identi- 
fied as gifted. Nevertheless, the same pat- 
terns of relationships appear to be present. 
McRAT students, regardless of classifica- 
tion, generally outperformed the control 
group students. 

Data in Table 8 shows the comparison 



In general, both minority and non-mi- 
nority students benefited from receiving 
McRAT instruction. Three studies in Table 
9 clearly show that all students, minority 
and non-minority, performed better after 
McRAT instruction than they did before. 
In general, the non-minority students 
showed slightly greater gains than did the 
minority students, although in all but two 
of these cases, the average gains in both 
groups were substantial. Two cases in 1990- 
91 in which the minority students' gains 
were markedly smaller than the non-mi- 
nority students warrant further examina- 
tion. 
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WHh 


WHhoat 


Analysis 


OYn. 


1 Yt 


lYr». 






Pre Post Gain 


Pre Post Gain 


Pre Pott Gam 


Pre Poet Gain Pre 


Pom Gain 


Analyst* 


511 63.3 11 2 


39.3 60.7 21.4 


U2 60.3 16.1 


535 645 11.0 605 


69.6 9.4 


Comparison 


41.7 56.1 14.4 


30.6 585 27.9 


Comparison 


27.9 51.7 23.8 


40.4 54.8 14.4 417 


59.9 17.2 




Inference 


42.4 59,6 17.2 


28.5 53.3 24.8 


Inference 


33.6 56.5 219 


39.0 60.0 21.0 48.1 


77.4 29.3 


Evaluation 


47.7 73.0 25.3 


39.6 68.9 29.3 


Evaluation 


44.3 71.3 27.0 


50.2 74.8 24.6 6U 


83.2 21.9 


TABLE 7 Cor 
19S9-90 for 


npamon of Mean Score 
McRAT and Control Gr 


s by Classification 
oup Students 


TABLE ( Compa 
Yearac 


nton of Mean Scores by ( 
f Previous McRAT Instru 


nassification for Stud 
etion 


!ntswithttl,or2 


1 <!,.,.(.,( 


McRAT 


Control 


im-92 


0Yr» 


1 Yr. 


2Yrs. 


Analysis 


Pre Post Cain 


Pre Post Gain 


Pre Post Gain 


Pre Pcsf. Gain 


Pre Post Gain | 


24.4 51 0 27 5 


41.4 47.9 65 


Analysis 
Companion 
Inference 
Evaluation 


33.1 56.0 22.9 


43.7 55.3 11.6 


58.8 75.5 16.7 


Companion 


ION 418 240 


204 37.1 16.7 


25.3 44.6 19.3 


36.8 49.6 1 2.8 


33.0 60.5 27.5 


Inference 


21.3 15.5 6.2 


20 7 10.7 1.0 


31.6 42.2 10.6 


35.4 53.2 17.8 


45.0 81.0 36.0 


Evaluation 


24 1 421) 170 


411.4 17.3 -3.1 


35.1 58.6 235 


45.1 65.8 20.7 


68.5 84.0 15.5 




44.1 50 7 6.6 


Analysis 
Companion 
Inference 
Evaluation 


44.1 60.6 16.5 


511 64.5 12.4 


58.2 675 9J 


Analysis 


15.0 25 0 


26.8 52.2 25.4 


39.8 55.1 15.3 


44.7 55.9 11.2 


Companion 


21 0 5| .1 2d 1 J 


25 6 .13 5 7.0 


33.0 57.7 24.7 


34.9 56.6 21.7 


45.3 72.1 268 


Inference 


210 46 7 22 8 


11 2 100 -0 1 


444 72.5 28.1 


50.0 73.1 23.1 


55.4 80.4 25.0 


Evaluation 


16 H 561 |15 


46 5 41 2 1 1 


Analysis 
Companion 
1 nlr pence 
lA-^lualion 


48 4 65.2 16.8 


67.5 74.6 7.1 


68.3 69.8 1 5 




54 4 14 


32 0 58.8 25.9 


48.7 60.4 11.7 


48.0 70.3 22.3 


Analysis 


42 4 Ml 21" 


35 0 66 3 104 


51.6 81.1 20.5 


57.0 00.0 33.0 


Comparison 


14 5 l.k y 14 11 


15 4 41 H "4 


40 0 81.4 32.4 


60.2 "0.9 30.7 


68.8 Of 22.9 


Inference 


26.6 41 h 2HI 


17 1 17(1 Jl.1 


Analysis 

C'umparittin 

Inference 

Evaluation 


408 52.2 11.4 


49.3 61.6 14.3 


"Nosr*cMl Ed 
itvJfuu hid I Vtfjn 
■J pnviout McRAT 
uiikmrttca 


I'vaUfctfion 


16 1 !>' 1 5 


44" -12 6 1 


2.10 41.3 18.3 


20.2 51.8 22.6 




281 42 4 14 1 


37.3 55.5 1B.2 


406 57.3 167 


38.2 67.0 28,8 
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The overwhelming picture from the 
remainder of the data indicates that both 
the minority and non-minority students 
showed important improvement, and this' 
case notwithstanding, the slightly higher 
gain scores by the non-minority students 
does not seem to indicate any systematic 
cultural disparities. 



TABLE 9 

1989-90 Comparison of Mean Scores for Minority and 
Non-Minority Students in the McRAT Group 



TABLE 10 Inter-Rater Reliability 



Finding 5 

The assessment procedures showed a 
high degree of inter-rater reliability. 



The inter-rater reliability data pre- 
sented in Table 10 are evidence of the time 
and effort spent by the McRAT program 
staff in :rainingand supervising the teach- 
ers who were used in. the scoring of the 
essays. In each year the large majority of 
skills, which were scored on each essay by 
multiple raters, were in close, if not per- 
fect, agreement. Grading of essays has long 
been considered as too subjective to do re- 
liably. The percentage of overall agreement 
and the large correlations indicate that 
teachers who have been trained in the 
McRAT curriculum can independently and 
reliably assess the quality of the students' 
performance. 

Discussion 

The Multicultural Reading and Think- 
ing Program (McRAT) is consistent with 
national goals for education in that stu- 
dents demonstrate the ability to reason, 
solve problems, apply knowledge, write 
and communicate effectively. It reflects the 
major shift in states' efforts in educational 
reform, focusing on the "core technology" 
of the school, classroom instruction. It fur- 
ther embodies the movement toward ex- 
plicit linkages between instructional pro- 
grams and "alternative assessment." Con- 
siderable effort has been given to ensur- 
ing the technical quality of the assessment 
techniques used by McRAT. The program 
is beneficial to all students, including the 
educationally disadvantaged, and it pro- 
motes students' understanding and appre- 
ciation of the diverse cultural heritage of 
the United States and the world. 
Ongoing Research 

Additional data from 1989-92 studies 
are available. Analyses of 1992-93 data are 
incomplete but seem to shew the same 
trends as the previous years. A 1993-94 
study of junior high students with previ- 
ous McRAT instruction vs. without previ- 
ous McRAT instruction lias been initiated. 

Slrilisfinif amilvM-s <>f data Have been provided 
by Dmt Mundltom, PH.D., assistant pwfWMir, CVh- 
ter /or Uwnmx and Research, UniverstiVKf Arkan- 
sas at I title Rock, lanita Hoskyn is pnyram man- 
nger-readttig, Division of Schml and Staff Suppmi 
Services. Arkansas Department of Education (501/ 
9 2-4232). 
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Minority 


Non-Minority 


Analysis 


Pre Post Cain 


Pre Post Gain 


33.9 55.9 22.0 


36.2 61.0 24.8 


Comparison 


21.2 44.5 233 


25.6 55.4 29.8 


Inference 


20.1 40.0 19.9 


25.9 47.4 21.5 


Evaluation 


31.3 47.5 16.2 


37.2 59.2 22.0 


1990-91 Comparison of Mean Scores for Minority 
and Non-Minority Students 


Analysis 


Minority 


Non-Minority 


Pre Post Gain 


Pre Post Gain 


50.3 57.3 7.0 


47.4 63.6 16.2 


Comparison 


37.2 52.3 15.1 


38.4 58.8 20.4 


Inference 


39.5 45.8 6.3 


38.5 59.7 21.2 


Evaluation 


43.6 63.2 19.6 


44.1 72.8 28.7 


1991-92 Comparison of Mean Scores for Minority 
and Non-Minority Students 


Analysis 


Minority 


Non-Minority 


Pro Post Gain 


Pro Post Cain 


46.7 60.0 13.3 


47.7 62.0 14 3 


Comparison 


31.0 50.4 19.4 


31.9 53.4 21.5 


Inference 


33.8 51.6 17.8 


36.7 60.2 23.5 


Evaluation 


42.7 68.6 25.9 


47.7 73.7 26.0 



Difference Frequeacy 


Percentage 


<4 


906 


754 


4 


141 


.118 


6 


90 


.075 


8 


33 


.028 


10 


10 


.008 


>10 


21 


— m7~ 


Total 


1200 


1 .000 


Difference 


Frequency 


Percentage 


0 


3465 


711 


*■ 


1065 


222 


4 


178 


.037 


6 


116 


.024 


8 


16 


.003 


10 


6 


□01 


12 


9 


.002 


14 


1 


000 


Total 


4176 


1 .000 




Difference 


Frequency 


Percentage 


II 


2817 


.66? 


2 


1049 


248 


4 


212 


050 


<! 


113 


.027 


Ji 


23 


.005 


10 


4 


.001 


12 


4 


.001 


14 


3 


.001 


IS 


1 


.000 


Total 


422* 


1 .000 



1989-90 

Overall 
Correlation 
Between 
Rater 1 and 
Rater 2: .80 



1990-91 
OveMll 
Correlation 
Bet wren 
Rater I and 
Rater 2: .95 



1991-92 
Overall 
Correlation 
Between 
Rater"! and 
Rater 2: .°4 
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